Responding to Representations of Physical Elements

ABSTRACT

In some implementations, a method includes obtaining, by a virtual intelligent agent (VIA), a perceptual property vector (PPV) for a graphical representation of a physical element. In some implementations, the PPV includes one or more perceptual characteristic values characterizing the graphical representation of the physical element. In some implementations, the method includes instantiating a graphical representation of the VIA in a graphical environment that includes the graphical representation of the physical element and an affordance that is associated with the graphical representation of the physical element. In some implementations, the method includes generating, by the VIA, an action for the graphical representation of the VIA based on the PPV. In some implementations, the method includes displaying a manipulation of the affordance by the graphical representation of the VIA in order to effectuate the action generated by the VIA.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of Intl. Patent App. No.PCT/US2020/28966, filed on Apr. 20, 2020, which claims priority to U.S.Provisional Patent App. No. 62/837,287, filed on Apr. 23, 2019, whichare both hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure generally relates to responding torepresentations of physical elements.

BACKGROUND

Some devices are capable of generating and presenting environments. Somedevices that present environments include mobile communication devicessuch as smartphones. Most previously available devices that present anenvironment are ineffective at allowing a user to interact with theenvironment.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood by those of ordinaryskill in the art, a more detailed description may be had by reference toaspects of some illustrative implementations, some of which are shown inthe accompanying drawings.

FIGS. 1A-1D are diagrams illustrating a virtual intelligent agent (VIA)interacting with graphical representations of physical elements inaccordance with some implementations.

FIG. 2 is a block diagram of an example device in accordance with someimplementations.

FIGS. 3A-3B are flowchart representations of a method of detecting andinteracting with XR representations of physical elements in accordancewith some implementations.

FIG. 4 is a block diagram of a device enabled with various componentsthat enable a VIA to detect and interact with XR representations ofphysical elements in accordance with some implementations.

In accordance with common practice the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, some of the drawings may not depict all of the componentsof a given system, method or device. Finally, like reference numeralsmay be used to denote like features throughout the specification andfigures.

SUMMARY

Various implementations disclosed herein include devices, systems, andmethods that enable a virtual intelligent agent (VIA) (e.g., anintelligent agent (IA)) to detect and interact with graphicalrepresentations of physical elements. In various implementations, adevice includes a non-transitory memory and one or more processorscoupled with the non-transitory memory. In some implementations, amethod includes obtaining, by a virtual intelligent agent (VIA), aperceptual property vector (PPV) for a graphical representation of aphysical element. In some implementations, the PPV includes one or moreperceptual characteristic values characterizing the graphicalrepresentation of the physical element. In some implementations, themethod includes instantiating a graphical representation of the VIA in agraphical environment that includes the graphical representation of thephysical element and an affordance that is associated with the graphicalrepresentation of the physical element. In some implementations, themethod includes generating, by the VIA, an action for the graphicalrepresentation of the VIA based on the PPV. In some implementations, themethod includes displaying a manipulation of the affordance by thegraphical representation of the VIA to effectuate the action generatedby the VIA.

In accordance with some implementations, a device includes one or moreprocessors, a non-transitory memory, and one or more programs. In someimplementations, the one or more programs are stored in thenon-transitory memory and are executed by the one or more processors. Insome implementations, the one or more programs include instructions forperforming or causing performance of any of the methods describedherein. In accordance with some implementations, a non-transitorycomputer readable storage medium has stored therein instructions that,when executed by one or more processors of a device, cause the device toperform or cause performance of any of the methods described herein. Inaccordance with some implementations, a device includes one or moreprocessors, a non-transitory memory, and means for performing or causingperformance of any of the methods described herein.

DESCRIPTION

Numerous details are described in order to provide a thoroughunderstanding of the example implementations shown in the drawings.However, the drawings merely show some example aspects of the presentdisclosure and are therefore not to be considered limiting. Those ofordinary skill in the art will appreciate that other effective aspectsand/or variants do not include all of the specific details describedherein. Moreover, well-known systems, methods, components, devices andcircuits have not been described in exhaustive detail so as not toobscure more pertinent aspects of the example implementations describedherein.

A physical environment refers to a physical world that people can senseand/or interact with without aid of electronic devices. The physicalenvironment may include physical features such as a physical surface ora physical object. For example, the physical environment corresponds toa physical park that includes physical trees, physical buildings, andphysical people. People can directly sense and/or interact with thephysical environment such as through sight, touch, hearing, taste, andsmell. In contrast, an extended reality (XR) environment refers to awholly or partially simulated environment that people sense and/orinteract with via an electronic device. For example, the XR environmentmay include augmented reality (AR) content, mixed reality (MR) content,virtual reality (VR) content, and/or the like. With an XR system, asubset of a person's physical motions, or representations thereof, aretracked, and, in response, one or more characteristics of one or morevirtual objects simulated in the XR environment are adjusted in a mannerthat comports with at least one law of physics. As one example, the XRsystem may detect head movement and, in response, adjust graphicalcontent and an acoustic field presented to the person in a mannersimilar to how such views and sounds would change in a physicalenvironment. As another example, the XR system may detect movement ofthe electronic device presenting the XR environment (e.g., a mobilephone, a tablet, a laptop, or the like) and, in response, adjustgraphical content and an acoustic field presented to the person in amanner similar to how such views and sounds would change in a physicalenvironment. In some situations (e.g., for accessibility reasons), theXR system may adjust characteristic(s) of graphical content in the XRenvironment in response to representations of physical motions (e.g.,vocal commands).

There are many different types of electronic systems that enable aperson to sense and/or interact with various XR environments. Examplesinclude head mountable systems, projection-based systems, heads-updisplays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmountable system may have one or more speaker(s) and an integratedopaque display. Alternatively, a head mountable system may be configuredto accept an external opaque display (e.g., a smartphone). The headmountable system may incorporate one or more imaging sensors to captureimages or video of the physical environment, and/or one or moremicrophones to capture audio of the physical environment. Rather than anopaque display, a head mountable system may have a transparent ortranslucent display. The transparent or translucent display may have amedium through which light representative of images is directed to aperson's eyes. The display may utilize digital light projection, OLEDs,LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, orany combination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In some implementations, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface.

The present disclosure provides methods, systems, and/or devices thatenable a graphical representation of a virtual intelligent agent (VIA)to detect and interact with graphical representations of physicalelements. Many physical elements do not have electronic transceiversthat emit data which identifies the physical elements. Such physicalelements are sometimes referred to as passive physical elements. Aperceptual property vector (PPV) for a graphical representation of aphysical element includes perceptual characteristic values thatcharacterize the graphical representation of the physical element.Populating a potentially detectable set of the VIA with the perceptualcharacteristic values allows the graphical representation of the VIA todetect and interact with the graphical representation of the physicalelement. The VIA utilizes the perceptual characteristic values includedin the PPV to generate an action which involves an interaction betweenthe graphical representation of the VIA and the graphical representationof the physical element. The graphical representation of the physicalelement is associated with an affordance. The graphical representationof the VIA manipulates the affordance associated with the graphicalrepresentation of the physical element to effectuate the actiongenerated by the VIA. Hereinafter, graphical environments are referredto as XR environments and graphical representations are referred to asXR representations for the sake of brevity.

FIG. 1A is a block diagram of an example operating environment 10 inaccordance with some implementations. While pertinent features areshown, those of ordinary skill in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity and so as not to obscure more pertinent aspectsof the example implementations disclosed herein. To that end, as anon-limiting example, the operating environment 10 includes anelectronic device 100.

In the example of FIG. 1A, the electronic device 100 is held by a user(not shown). In some implementations, the electronic device 100 includesa smartphone, a tablet, a laptop, or the like. In some implementations,the electronic device 100 includes a wearable computing device that isworn by the user. For example, in some implementations, the electronicdevice 100 includes a head-mountable device (HMD). In someimplementations, the HMD is shaped to form a receptacle that receives adevice with a display (e.g., the device with the display can be slidinto the HMD to serve as a display for the HMD). Alternatively, in someimplementations, the HMD includes an integrated display.

In various implementations, the electronic device 100 includes a virtualintelligent agent (VIA) 110. In various implementations, the VIA 110performs an action in order to satisfy (e.g., complete or achieve) anobjective of the VIA 110. In various implementations, the VIA 110obtains the objective from a human operator (e.g., the user of theelectronic device 100). For example, in some implementations, the VIA110 generates responses to queries that the user of the electronicdevice 100 inputs into the electronic device 100. In someimplementations, the VIA 110 synthesizes vocal responses to voicequeries that the electronic device 100 detects. In variousimplementations, the VIA 110 performs electronic operations on theelectronic device 100. For example, the VIA 110 composes messages inresponse to receiving an instruction from the user of the electronicdevice 100. In some implementations, the VIA 110 schedules calendarevents, sets timers/alarms, provides navigation directions, readsincoming messages, and/or assists the user in operating the electronicdevice 100. In various implementations, the VIA 110 is referred to as anintelligent agent (IA).

In various implementations, the electronic device 100 presents anextended reality (XR) environment 120. In some implementations, theelectronic device 100 receives the XR environment 120 from anotherdevice. In some implementations, the electronic device 100 retrieves theXR environment 120 from a non-transitory memory (e.g., from a remotedata source). In some implementations, the electronic device 100generates the XR environment 120. For example, in some implementations,the electronic device 100 synthesizes the XR environment 120 based on asemantic construction of a physical environment. In variousimplementations, the XR environment 120 corresponds to a physicalenvironment. In some implementations, the XR environment 120 is within adegree of similarity to a corresponding physical environment.

In the example of FIG. 1A, the XR environment 120 includes XRrepresentations of physical elements 122. In some implementations, theXR representations of physical elements 122 correspond to respectivephysical elements in a physical environment. In such implementations,the XR representations of physical elements 122 are within a degree ofsimilarity to their corresponding physical elements. In the example ofFIG. 1A, the XR environment 120 includes XR representations of boundingsurfaces 124. In some implementations, the XR representations ofbounding surfaces 124 correspond to respective physical surfaces in thephysical environment. In such implementations, the XR representations ofbounding surfaces 124 are within a degree of similarity to theircorresponding physical surfaces.

In the example of FIG. 1A, the XR environment 120 includes an XRtelevision 122 a, an XR couch 122 b, an XR coffee table 122 c, an XRtelevision remote 122 d, an XR door 122 e and an XR door handle 122 f.In some implementations, the XR television 122 a, the XR couch 122 b,the XR coffee table 122 c, the XR television remote 122 d, the XR door122 e and the XR door handle 122 f represent a real television, a realcouch, a real coffee table, a real television remote, a real door and areal door handle, respectively, located in a physical environmentrepresented by the XR environment 120.

In various implementations, XR properties of the XR representations ofphysical elements 122 are within a degree of similarity to physicalproperties of corresponding physical elements located in the physicalenvironment. In some implementations, visual properties of the XRrepresentations of physical elements 122 are selected to match visualproperties of the corresponding physical elements located in thephysical environment. For example, a color of the XR couch 122 b iswithin a degree of similarity to a color of the corresponding realcouch. Similarly, a texture of the XR coffee table 122 c is within adegree of similarity to a texture of the corresponding real coffeetable.

In the example of FIG. 1A, the XR representations of bounding surfaces124 include an XR floor 124 a, an XR front wall 124 b and an XR sidewall 124 c. In some implementations, the XR floor 124 a, the XR frontwall 124 b and the XR side wall 124 c represent a real floor, a realfront wall and a real side wall, respectively, of a physical environmentrepresented by the XR environment 120. As such, the XR representationsof bounding surfaces 124 are within a degree of similarity to physicalsurfaces in the physical environment.

In various implementations, the electronic device 100 obtains respectiveperceptual property vectors (PPVs) 130 for the XR representations ofphysical elements 122 and the XR representations of bounding surfaces124. In various implementations, each PPV 130 includes one or moreperceptual characteristic values 132 characterizing a corresponding XRrepresentation of a physical element. For example, the PPVs 130 includea first PPV which includes a first set of perceptual characteristicvalues that characterize the XR television 122 a (e.g., the first set ofperceptual characteristic values indicate a size of the XR television122 a, a resolution of the XR television 122 a, a refresh rate of the XRtelevision 122 a, etc.). Similarly, the PPVs 130 include a second PPVwhich includes a second set of perceptual characteristic values thatcharacterize the XR couch 122 b (e.g., the second set of perceptualcharacteristic values indicate a size, a color, a texture and/or amaterial of the XR couch 122 b).

In various implementations, the perceptual characteristic values 132characterize one or more physical properties of the XR representationsof the physical elements 122. In some implementations, the perceptualcharacteristic values 132 characterize a texture of the XRrepresentation. For example, the perceptual characteristic values 132for an XR representation indicate whether the XR representation appearssmooth or rough when touched by an XR object such as an XRrepresentation of the VIA 110.

In some implementations, the perceptual characteristic values 132characterize a hardness of the XR representation of the physicalelement. For example, the perceptual characteristic values 132 for theXR couch 122 b characterize a hardness of an arm rest and/or a hardnessof a cushion of the XR couch 122 b. As another example, the perceptualcharacteristic values 132 for the XR floor 124 a characterize a hardnessof the XR floor 124 a, which determines the result of dropping an XRobject on the XR floor 124 a. For example, if the perceptualcharacteristic values 132 for the XR floor 124 a indicate that the XRfloor 124 a is as hard as concrete then dropping a delicate XR objectsuch as a glass may result in the XR object breaking. However, if theperceptual characteristic values 132 for the XR floor 124 a indicatethat the XR floor 124 a is as soft as carpet then dropping the delicateXR object may result in the XR object staying intact.

In various implementations, the perceptual characteristic values 132characterize a smell of the XR representation of the physical element.For example, in some implementations, the perceptual characteristicvalues 132 define an odor function for the XR representation of thephysical element. As an example, the perceptual characteristic values132 for the XR couch 122 b characterize how the XR couch 122 b smells toan XR object such as an XR dog or an XR human.

Referring to FIG. 1B, in some implementations, the XR environment 120includes an XR representation 126 of the VIA 110. In the example of FIG.1B, the XR representation 126 of the VIA 110 includes an XR human. Insome implementations, a user of the electronic device 100 selects the XRrepresentation 126 for the VIA 110 from a set of available XRrepresentations. In various implementations, the XR representation 126of the VIA 110 is customizable. For example, in some implementations,the XR representation 126 of the VIA 110 includes an XR dog, an XRrobot, etc.

In various implementations, the XR representation 126 of the VIA 110performs an action within the XR environment 120 in order to satisfy(e.g., complete or achieve) an objective of the VIA 110. In someimplementations, the VIA 110 obtains the objective from a human operator(e.g., a user of the electronic device 100). In some implementations,the XR representation 126 of the VIA 110 obtains the objective from anXR representation of the human operator. For example, the XRrepresentation of the human operator instructs the XR representation 126of the VIA 110 to perform an action in the XR environment 120.

In various implementations, the VIA 110 performs an action or causesperformance of the action by manipulating the XR representation 126 ofthe VIA 110 in the XR environment 120. In some implementations, the XRrepresentation 126 of the VIA 110 is able to perform XR actions that theXR representation of the human operator is incapable of performing. Insome implementations, the XR representation 126 of the VIA 110 performsXR actions based on information that the VIA 110 obtains from a physicalenvironment. For example, the XR representation 126 of the VIA 110nudges the XR representation of the human operator when the VIA 110detects ringing of a doorbell in the physical environment.

Referring to FIG. 1C, in various implementations, the XR representation126 of the VIA 110 is associated with a potentially detectable set 112.In some implementations, the potentially detectable set 112 includes XRrepresentations of physical elements that the XR representation 126 ofthe VIA 110 can detect (e.g., see, hear and/or smell). For example, thepotentially detectable set 112 includes at least some of the XRrepresentations of physical elements 122 in the XR environment 120. Insome implementations, the potentially detectable set 112 includesperceptual characteristics values for various XR representations ofphysical elements that the XR representation 126 of the VIA 110 candetect.

In some implementations, the potentially detectable set 112 includes apotentially visible subset, a potentially audible subset and apotentially smellable subset. The potentially visible subset includesvisual properties of the XR representations of physical elements 122that the XR representation 126 of the VIA 110 can see (e.g., a displayscreen of the XR television 122 a, a surface of the XR couch 122 b,etc.). The potentially audible subset includes audible properties of theXR representations of physical elements 122 that the XR representation126 of the VIA 110 can hear (e.g., sounds emitted by the XR television122 a, and sounds made by the XR door 122 e when the XR door 122 eopens/closes). The potentially smellable subset includes smellproperties (e.g., olfaction properties) of the XR representations ofphysical elements 122 that the XR representation 126 of the VIA 110 cansmell (e.g., an odor of the XR couch 122 b).

In some implementations, the VIA 110 populates the potentiallydetectable set 112 based on the perceptual characteristic values 132included in the PPVs 130. For example, the VIA 110 populates thepotentially detectable set 112 with an odor function of the XR couch 122b in order to allow the XR representation 126 of the VIA 110 to smell anodor of the XR couch 122 b. Populating the potentially detectable set112 based on the PPVs 130 of the XR representations of physical elements122 allows the XR representation 126 of the VIA 110 to detect andinteract with the XR representations of physical elements 122.

In various implementations, the VIA 110 generates an action 114 for theXR representation 126 of the VIA 110 based on the PPV(s) 130. In someimplementations, the action 114 includes detecting and/or interactingwith one or more of the XR representations of physical elements 122. Forexample, in some implementations, the action 114 includes turning ON theXR television 122 a, jumping on the XR couch 122 b, opening the XR door122 e, etc.

Referring to FIG. 1D, in various implementations, the electronic device100 displays respective affordances 140 in association with the XRrepresentations of physical elements 122. For example, the electronicdevice 100 composites a television affordance 140 a in association withthe XR television 122 a, a couch affordance 140 b in association withthe XR couch 122 b, a coffee table affordance 140 c in association withthe XR coffee table 122 c, a television remote affordance 140 d inassociation with the XR television remote 122 d, a door affordance 140 ein association with the XR door 122 e, and a door handle affordance 140f in association with the XR door handle 122 f.

In various implementations, the affordances 140 allow interaction withthe corresponding XR representation of physical elements 122 inaccordance with the perceptual characteristic values 132 included intheir corresponding PPVs 130. For example, the television affordance 140a allows interaction with the XR television 122 a in accordance with theperceptual characteristic values 132 included in the PPV 130 for the XRtelevision 122 a (e.g., the XR representation 126 of the VIA 110 canactivate the television affordance 140 a to turn the XR television 122 aON or OFF). Similarly, the door handle affordance 140 f allowsinteraction with the XR door handle 122 f in accordance with theperceptual characteristic values 132 included in the PPV 130 for the XRdoor handle 122 f (e.g., the XR representation 126 of the VIA 110 caninvoke the door handle affordance 140 f to turn the XR door handle 122f).

In some implementations, the action 114 includes activating one or moreof the affordances 140 to interact with the corresponding XRrepresentation of physical elements 122. For example, in someimplementations, the action 114 includes causing the XR representation126 of the VIA 110 to move closer to the door handle affordance 140 fand manipulating (e.g., activating) the door handle affordance 140 f inorder to turn the XR door handle 122 f which can result inopening/closing of the XR door 122 e. Similarly, in someimplementations, the action 114 includes causing the XR representation126 of the VIA 110 to move closer to the television remote affordance140 d and manipulating the television remote affordance 140 d in orderto pick-up the XR television remote 122 d. After picking-up thetelevision remote affordance 140 d, the action 114 can cause the XRrepresentation 126 of the VIA 110 to manipulate the television remoteaffordance 140 d again in order to operate the XR television 122 a viathe XR television remote 122 d. In the example of FIG. 1D, the XRrepresentation 126 of the VIA 110 is manipulating the televisionaffordance 140 a, for example, because the action 114 is to turn the XRtelevision 122 a ON or OFF. More generally, in various implementations,the electronic device 100 displays a manipulation of one of theaffordances 140 by the XR representation 126 of the VIA 110 in order toeffectuate the action 114 generated by the VIA 110.

In some implementations, a head-mountable device (HMD) (not shown),being worn by a user, presents (e.g., displays) the XR environment 120according to various implementations. In some implementations, the HMDincludes an integrated display (e.g., a built-in display) that displaysthe XR environment 120. In some implementations, the HMD includes ahead-mountable enclosure. In various implementations, the head-mountableenclosure includes an attachment region to which another device with adisplay can be attached. For example, in some implementations, theelectronic device 100 can be attached to the head-mountable enclosure.In various implementations, the head-mountable enclosure is shaped toform a receptacle for receiving another device that includes a display(e.g., the electronic device 100). For example, in some implementations,the electronic device 100 slides/snaps into or otherwise attaches to thehead-mountable enclosure. In some implementations, the display of thedevice attached to the head-mountable enclosure presents (e.g.,displays) the XR environment 120.

FIG. 2 illustrates a block diagram of an electronic device 200. In someimplementations, the electronic device 200 implements the electronicdevice 100 shown in FIGS. 1A-1D. As illustrated in FIG. 2, in someimplementations, the electronic device 200 includes a data obtainer 210,an action generator 220, and an XR environment generator 230.

In various implementations, the data obtainer 210 obtains the PPVs 130for the XR representations of physical elements 122. As describedherein, the PPVs 130 include respective perceptual characteristic values132 characterizing the XR representations of physical elements 122. Insome implementations, the data obtainer 210 retrieves the PPVs 130 froma non-transitory memory of the electronic device 200, or from a remotedata source. In some implementations, the data obtainer 210 receives thePPVs 130 from another device that generated by the PPVs 130. In someimplementations, the data obtainer 210 generates the PPVs 130 based oninformation encoded in a semantic construction of a physicalenvironment. In some implementations, the data obtainer 210 provides thePPVs 130 to the action generator 220 and/or the XR environment generator230.

In various implementations, the action generator 220 generates theaction 114 based on the PPV(s) 130. In some implementations, the action114 is for the XR representation 126 of the VIA 110 shown in FIGS. 1Cand 1D. In some implementations, the action generator 220 includes aneural network system that accepts the PPV(s) 130 and/or the perceptualcharacteristic values 132 as input(s) and outputs the action 114. Insome implementations, the action 114 includes detecting one of the XRrepresentations of physical elements 122 based on their correspondingperceptual characteristic values 132. In some implementations, theaction 114 includes interacting with one of the XR representations ofphysical elements 122 based on their corresponding perceptualcharacteristic values 132. In some implementations, the action generator220 provides the action 114 to the XR environment generator 230.

In various implementations, the XR environment generator 230 presentsthe XR environment 120. The XR environment generator 230 also displaysthe XR representations of physical elements 122, the XR representation126 of the VIA 110, and the affordances 140 in association with the XRrepresentations of physical elements 122. In some implementations, theXR environment generator 230 displays a manipulation of one of theaffordances 140 associated with one of the XR representations ofphysical elements 122 in order to effectuate the action 114 generated bythe VIA 110.

In some implementations, the XR environment generator 230 causes the XRrepresentation 126 of the VIA 110 to move closer to the affordance 140that is to be manipulated. After the XR representation 126 of the VIA110 is adjacent to the affordance 140 that is to be manipulated, the XRenvironment generator 230 causes the XR representation 126 of the VIA110 to manipulate (e.g., activate) the affordance 140.

FIG. 3A is a flowchart representation of a method 300 of detecting andinteracting with XR representations of physical elements in accordancewith some implementations. In various implementations, the method 300 isperformed by a device with a non-transitory memory and one or moreprocessors coupled with the non-transitory memory (e.g., the electronicdevice 100 shown in FIGS. 1A-1D and/or the electronic device 200 shownin FIG. 2). In some implementations, the method 300 is performed byprocessing logic, including hardware, firmware, software, or acombination thereof. In some implementations, the method 300 isperformed by a processor executing code stored in a non-transitorycomputer-readable medium (e.g., a memory).

As represented by block 310, in some implementations, the method 300includes obtaining, by a virtual intelligent agent (VIA), a perceptualproperty vector (PPV) for an XR representation of a physical element.For example, as shown in FIG. 1A, the VIA 110 obtains the PPVs 130 forthe XR representations of physical elements 122. In someimplementations, the PPV includes one or more perceptual characteristicvalues characterizing the XR representation of the physical element. Forexample, as shown in FIG. 1A, each PPV 130 includes one or moreperceptual characteristic values 132 characterizing a corresponding oneof the XR representations of the physical elements 122.

As represented by block 320, in some implementations, the method 300includes instantiating an XR representation of the VIA in an XRenvironment that includes the XR representation of the physical elementand an affordance that is associated with the XR representation of thephysical element. For example, as shown in FIG. 1D, the XR environment120 includes the XR representation 126 of the VIA 110, the XRrepresentations of physical elements 122, and the affordances 140 thatare associated with the XR representations of physical elements 122.

As represented by block 330, in some implementations, the method 300includes generating, by the VIA, an action for the XR representation ofthe VIA based on the PPV. For example, as shown in FIGS. 1C and 1D, theVIA 110 generates the action 114 for the XR representation 126 of theVIA 110 based on the PPV(s) 130.

As represented by block 340, in some implementations, the method 300includes displaying a manipulation of the affordance by the XRrepresentation of the VIA in order to effectuate the action generated bythe VIA. For example, as shown in FIG. 1D, the XR representation 126 ofthe VIA 110 is manipulating the television affordance 140 a in order toeffectuate the action 114 with respect to the XR television 122 a (e.g.,in order to control the XR television 122 a, for example, in order toturn the XR television 122 a ON or OFF).

Referring to FIG. 3B, as represented by block 310 a, in someimplementations, the method 300 includes populating a potentiallydetectable set of the VIA based on the PPV. For example, as shown inFIG. 1C, the VIA 110 populates the potentially detectable set 112 of theXR representation 126 of the VIA 110 based on the PPV(s) 130. In someimplementations, populating the potentially detectable set includespopulating a potentially visible subset of the VIA based on the PPV. Insome implementations, populating the potentially detectable set includespopulating a potentially audible subset of the VIA based on the PPV. Insome implementations, populating the potentially detectable set includespopulating a potentially smellable subset of the VIA based on the PPV.

As represented by block 310 b, in some implementations, populating thepotentially detectable set of the VIA allows the XR representation ofthe VIA to detect and/or interact with the XR representation of thephysical element. For example, populating the potentially detectable set112, shown in FIGS. 1C and 1D, allows the XR representation 126 of theVIA 110 to detect and/or interact with the XR representation of physicalelements 122.

As represented by block 310 b, in some implementations, populating thepotentially detectable set of the VIA allows the XR representation ofthe VIA to detect a texture of the XR representation of the physicalelement. For example, populating the potentially detectable set 112,shown in FIGS. 1C and 1D, with texture characteristics of the XR couch122 b allows the XR representation 126 of the VIA 110 to detect (e.g.,sense or feel) the texture of the XR couch 122 b.

As represented by block 310 b, in some implementations, populating thepotentially detectable set of the VIA allows the XR representation ofthe VIA to detect a hardness of the XR representation of the physicalelement. For example, populating the potentially detectable set 112,shown in FIGS. 1C and 1D, with hardness characteristics of the XR coffeetable 122 c allows the XR representation 126 of the VIA 110 to detect(e.g., sense or feel) the hardness of the XR coffee table 122 c.

As represented by block 310 b, in some implementations, populating thepotentially detectable set of the VIA allows the XR representation ofthe VIA to detect a smell associated with the XR representation of thephysical element. For example, populating the potentially detectable set112, shown in FIGS. 1C and 1D, with smell characteristics (e.g., an odorfunction) of the XR couch 122 b allows the XR representation 126 of theVIA 110 to detect (e.g., smell) the odor of the XR couch 122 b.

As represented by block 310 b, in some implementations, the XRrepresentation of the VIA detects a degree of the smell based on adistance between the XR representation of the VIA and the XRrepresentation of the physical element. For example, populating thepotentially detectable set 112, shown in FIGS. 1C and 1D, with an odorfunction of the XR couch 122 b allows the XR representation 126 of theVIA 110 to detect (e.g., smell) the odor of the XR couch 122 b withvarying degrees based on a distance between the XR representation 126 ofthe VIA 110 and the XR couch 122 b.

As represented by block 310 c, in some implementations, the method 300includes receiving the PPV from another device that generated the PPV.As represented by block 310 d, in some implementations, the method 300includes retrieving the PPV from the non-transitory memory or a remotedata source.

As represented by block 330 a, in some implementations, the actionincludes the XR representation of the VIA touching the XR representationof the physical element. For example, in some implementations, theaction 114, shown in FIG. 1D, includes the XR representation 126 of theVIA 110 touching the XR couch 122 b by manipulating the couch affordance140 b.

As represented by block 330 a, in some implementations, the actionincludes the XR representation of the VIA picking-up the XRrepresentation of the physical element. For example, in someimplementations, the action 114, shown in FIG. 1D, includes the XRrepresentation 126 of the VIA 110 picking-up the XR television remote122 d by manipulating the television remote affordance 140 d.

As represented by block 330 a, in some implementations, the actionincludes the XR representation of the VIA modifying the XRrepresentation of the physical element. For example, in someimplementations, the action 114, shown in FIG. 1D, includes the XRrepresentation 126 of the VIA 110 modifying the XR television remote 122d (e.g., by removing XR batteries from the XR television remote 122 d)by manipulating the television remote affordance 140 d.

As represented by block 330 a, in some implementations, the actionincludes the XR representation of the VIA breaking the XR representationof the physical element. For example, in some implementations, theaction 114, shown in FIG. 1D, includes the XR representation 126 of theVIA 110 breaking the XR coffee table 122 c by manipulating the coffeetable affordance 140 c.

As represented by block 330 b, in some implementations, the actionincludes the XR representation of the VIA changing a state of the XRrepresentation of the physical element. For example, in someimplementations, the action 114, shown in FIG. 1D, includes the XRrepresentation 126 of the VIA 110 opening/closing the XR door 122 e bymanipulating the door affordance 140 e.

As represented by block 330 c, in some implementations, the actionincludes the XR representation of the VIA smelling an odor associatedwith (e.g., emanating from) the XR representation of the physicalelement. For example, in some implementations, the action 114, shown inFIG. 1D, includes the XR representation 126 of the VIA 110 smelling anodor emanating from the XR couch 122 b.

As represented by block 330 d, in some implementations, the actionincludes the XR representation of the VIA hearing a sound generated by(e.g., emitted by) the XR representation of the physical element. Forexample, in some implementations, the action 114, shown in FIG. 1D,includes the XR representation 126 of the VIA 110 hearing soundsgenerated by the XR television 122 a.

As represented by block 350, in some implementations, the methodincludes obtaining, by the VIA, a second PPV for an XR representation ofa second physical element. For example, as shown in FIG. 1A, the VIA 110obtains respective PPVs 130 for the XR representations of physicalelements 122. The second PPV includes one or more perceptualcharacteristic values characterizing the XR representation of the secondphysical element. For example, as shown in FIG. 1A, each PPV 130includes a set of one or more perceptual characteristic values 132. Insome implementations, the method 300 includes displaying a secondaffordance that is associated with the XR representation of the secondphysical element. For example, as shown in FIG. 1D, the electronicdevice 100 displays respective affordances 140 in association with theXR representations of physical elements 122. In some implementations,the method 300 includes generating, by the VIA, a second action for theXR representation of the VIA based on the second PPV. For example, theaction 114 shown in FIGS. 1C and 1D includes multiple actions. In someimplementations, the method 300 includes displaying a manipulation ofthe second affordance by the XR representation of the VIA in order toeffectuate the second action generated by the VIA. For example, as shownin FIG. 1D, the XR representation 126 of the VIA 110 manipulates one ofthe affordances 140 in order to effectuate the action(s) 114.

FIG. 4 is a block diagram of a device 400 (e.g., the electronic device100 shown in FIGS. 1A-1D and/or the electronic device 200 shown in FIG.2) in accordance with some implementations. While certain specificfeatures are illustrated, those of ordinary skill in the art willappreciate from the present disclosure that various other features havenot been illustrated for the sake of brevity, and so as not to obscuremore pertinent aspects of the implementations disclosed herein. To thatend, as a non-limiting example, in some implementations the device 400includes one or more processing units (CPUs) 401, a network interface402, a programming interface 403, a memory 404, input/output (I/O)sensors 405 and one or more communication buses 406 for interconnectingthese and various other components.

In some implementations, the network interface 402 is provided to, amongother uses, establish and maintain a metadata tunnel between a cloudhosted network management system and at least one private networkincluding one or more compliant devices. In some implementations, theone or more communication buses 406 include circuitry that interconnectsand controls communications between system components. The memory 404includes high-speed random access memory, such as DRAM, SRAM, DDR RAM orother random access solid state memory devices, and may includenon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. The memory 404 optionallyincludes one or more storage devices remotely located from the one ormore CPUs 401. The memory 404 comprises a non-transitory computerreadable storage medium.

In some implementations, the I/O sensor 405 includes an image sensor(e.g., a camera) that captures images and/or videos of a physicalenvironment. In some implementations, the I/O sensor 405 includes adepth sensor that captures depth data for a physical environment.

In some implementations, the memory 404 or the non-transitory computerreadable storage medium of the memory 404 stores the following programs,modules and data structures, or a subset thereof including an optionaloperating system 408, the data obtainer 210, the action generator 220,and the XR environment generator 230. As described herein, in variousimplementations, the data obtainer 210 obtains a PPV (e.g., the PPVs 130shown in FIGS. 1A-2). To that end, the data obtainer 210 includesinstructions 210 a, and heuristics and metadata 210 b. As describedherein, in various implementations, the action generator 220 generatesan action based on the PPV (e.g., the action 114 shown in FIGS. 1C-2).To that end, the action generator 220 includes instructions 220 a, andheuristics and metadata 220 b. As described herein, in variousimplementations, the XR environment generator 230 displays amanipulation of the affordance by the XR representation of the VIA inorder to effectuate the action. To that end, the XR environmentgenerator 230 includes instructions 230 a, and heuristics and metadata230 b.

In some implementations, the VIA 110 shown in FIGS. 1A-1D includes anobjective-effectuator. In some implementations, an objective-effectuatorperforms an action in order to satisfy (e.g., complete or achieve) anobjective. In some implementations, an objective-effectuator isassociated with a particular objective, and the objective-effectuatorperforms actions that improve the likelihood of satisfying thatparticular objective. In some implementations, XR representations of theobjective-effectuators are referred to as object representations, forexample, because the XR representations of the objective-effectuatorsrepresent various objects (e.g., real objects, or fictional objects). Insome implementations, an objective-effectuator representing a characteris referred to as a character objective-effectuator. In someimplementations, a character objective-effectuator performs actions toeffectuate a character objective. In some implementations, anobjective-effectuator representing an equipment is referred to as anequipment objective-effectuator. In some implementations, an equipmentobjective-effectuator performs actions to effectuate an equipmentobjective. In some implementations, an objective-effectuatorrepresenting an environment is referred to as an environmentalobjective-effectuator. In some implementations, an environmentalobjective-effectuator performs environmental actions to effectuate anenvironmental objective.

While various aspects of implementations within the scope of theappended claims are described above, it should be apparent that thevarious features of implementations described above may be embodied in awide variety of forms and that any specific structure and/or functiondescribed above is merely illustrative. Based on the present disclosureone skilled in the art should appreciate that an aspect described hereinmay be implemented independently of any other aspects and that two ormore of these aspects may be combined in various ways. For example, anapparatus may be implemented and/or a method may be practiced using anynumber of the aspects set forth herein. In addition, such an apparatusmay be implemented and/or such a method may be practiced using otherstructure and/or functionality in addition to or other than one or moreof the aspects set forth herein.

It will also be understood that, although the terms “first”, “second”,etc. may be used herein to describe various elements, these elementsshould not be limited by these terms. These terms are only used todistinguish one element from another. For example, a first node could betermed a second node, and, similarly, a second node could be termed afirst node, which changing the meaning of the description, so long asall occurrences of the “first node” are renamed consistently and alloccurrences of the “second node” are renamed consistently. The firstnode and the second node are both nodes, but they are not the same node.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the claims. Asused in the description of the embodiments and the appended claims, thesingular forms “a”, “an”, and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It willalso be understood that the term “and/or” as used herein refers to andencompasses any and all possible combinations of one or more of theassociated listed items. It will be further understood that the terms“comprises” and/or “comprising” when used in this specification, specifythe presence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in accordance with a determination”or “in response to detecting” that a stated condition precedent is true,depending on the context. Similarly, the phrase “if it is determined[that a stated condition precedent is true]” or “if [a stated conditionprecedent is true]” or “when [a stated condition precedent is true]” maybe construed to mean “upon determining” or “in response to determining”or “in accordance with a determination” or “upon detecting” or “inresponse to detecting” that the stated condition precedent is true,depending on the context.

What is claimed is:
 1. A method comprising: at a device including anon-transitory memory and one or more processors coupled with thenon-transitory memory: obtaining, by an intelligent agent (IA), one ormore perceptual characteristic values characterizing a graphicalrepresentation of a physical element; instantiating a graphicalrepresentation of the IA in a graphical environment that includes thegraphical representation of the physical element and an affordance thatis associated with the graphical representation of the physical element;generating, by the IA, an action for the graphical representation of theIA based on the one or more perceptual characteristic values; anddisplaying a manipulation of the affordance by the graphicalrepresentation of the IA to effectuate the action generated by the IA.2. The method of claim 1, further comprising: populating a potentiallydetectable set of the IA based on the one or more perceptualcharacteristic values.
 3. The method of claim 2, wherein populating thepotentially detectable set of the IA allows the graphical representationof the IA to detect or interact with the graphical representation of thephysical element.
 4. The method of claim 2, wherein populating thepotentially detectable set of the IA allows the graphical representationof the IA to detect a texture of the graphical representation of thephysical element.
 5. The method of claim 2, wherein populating thepotentially detectable set of the IA allows the graphical representationof the IA to detect a hardness of the graphical representation of thephysical element.
 6. The method of claim 2, wherein populating thepotentially detectable set of the IA allows the graphical representationof the IA to detect a smell associated with the graphical representationof the physical element.
 7. The method of claim 6, wherein the graphicalrepresentation of the IA detects a degree of the smell based on adistance between the graphical representation of the IA and thegraphical representation of the physical element.
 8. The method of claim2, wherein populating the potentially detectable set comprises:populating a potentially visible subset of the IA based on the one ormore perceptual characteristic values.
 9. The method of claim 2, whereinpopulating the potentially detectable set comprises: populating apotentially audible subset of the IA based on the one or more perceptualcharacteristic values.
 10. The method of claim 2, wherein populating thepotentially detectable set comprises: populating a potentially smellablesubset of the IA based on the one or more perceptual characteristicvalues.
 11. The method of claim 1, wherein the action includes thegraphical representation of the IA detecting the graphicalrepresentation of the physical element.
 12. The method of claim 1,wherein the action includes the graphical representation of the IAtouching the graphical representation of the physical element.
 13. Themethod of claim 1, wherein the action includes the graphicalrepresentation of the IA picking-up the graphical representation of thephysical element.
 14. The method of claim 1, wherein the action includesthe graphical representation of the IA modifying the graphicalrepresentation of the physical element.
 15. The method of claim 1,wherein the action includes the graphical representation of the IAbreaking the graphical representation of the physical element.
 16. Themethod of claim 1, wherein the action includes the graphicalrepresentation of the IA changing a state of the graphicalrepresentation of the physical element.
 17. The method of claim 1,wherein the action includes the graphical representation of the IAsmelling an odor associated with the graphical representation of thephysical element.
 18. The method of claim 1, further comprising:obtaining, by the IA, one or more perceptual characteristic valuescharacterizing a graphical representation of a second physical element;displaying a second affordance that is associated with the graphicalrepresentation of the second physical element; generating, by the IA, asecond action for the graphical representation of the VIA based on theone or more perceptual characteristic values characterizing thegraphical representation of the second physical element; and displayinga manipulation of the second affordance by the graphical representationof the IA in order to effectuate the second action generated by the IA.19. A device comprising: one or more processors; a non-transitorymemory; one or more displays; and one or more programs stored in thenon-transitory memory, which, when executed by the one or moreprocessors, cause the device to: obtain, by an intelligent agent (IA),one or more perceptual characteristic values characterizing a graphicalrepresentation of a physical element; instantiate a graphicalrepresentation of the IA in a graphical environment that includes thegraphical representation of the physical element and an affordance thatis associated with the graphical representation of the physical element;generate, by the IA, an action for the graphical representation of theIA based on the one or more perceptual characteristic values; anddisplay a manipulation of the affordance by the graphical representationof the IA to effectuate the action generated by the IA.
 20. Anon-transitory memory storing one or more programs, which, when executedby one or more processors of a device with a display, cause the deviceto: obtain, by an intelligent agent (IA), one or more perceptualcharacteristic values characterizing a graphical representation of aphysical element; instantiate a graphical representation of the IA in agraphical environment that includes the graphical representation of thephysical element and an affordance that is associated with the graphicalrepresentation of the physical element; generate, by the IA, an actionfor the graphical representation of the IA based on the one or moreperceptual characteristic values; and display a manipulation of theaffordance by the graphical representation of the IA to effectuate theaction generated by the IA.